CS168: The Modern Algorithmic Toolbox Lecture #6: Stochastic Gradient Descent and Regularization

Last lecture we covered the basics of gradient descent, with an emphasis on the intuition behind and geometry underlying the method, plus a concrete instantiation of it for the problem of linear regression (fitting the best hyperplane to a set of data points). This basic method is already interesting and useful in its own right (see Homework #3). This lecture we’ll cover two extensions that, while simple, will bring your knowledge a step closer to the state-of-the-art in modern machine learning. The two extensions have different characters. The first concerns how to actually solve (computationally) a given unconstrained minimization problem, and gives a modification of basic gradient descent — “stochastic gradient descent” — that scales to much larger data sets. The second extension concerns problem formulation rather than implementation, namely the choice of the unconstrained optimization problem to solve (i.e., the objective function f). Here, we introduce the idea of “regularization,” with the goal of avoiding overfitting the function learned to the data set at hand, even for very high-dimensional data.